Creating Realistic Laboratory Settings: Comparative Studies of Three Think-Aloud Usability Evaluations of a Mobile System

This paper addresses the issue of creating a realistic laboratory setting when evaluating the usability of mobile systems. Three laboratory-based think-aloud evaluations of the same mobile system were designed and conducted for the purpose of comparing the impact of different approaches to creating realistic laboratory environments on the results subsequently produced. The three evaluations spanned the use of test subjects with and without domain specific knowledge and the use of low and high fidelity simulations of the use context. The results show similarities and differences in the results of the three evaluations. First, an equal number of usability problems are identified in the evaluations, and a standard laboratory involving non-domain subjects can identify severe problems. On the other hand, employing domain test subjects in a simulated laboratory seems to generate some additional problems which would be difficult to identify in the standard laboratory.